{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we build a simple matrix factorization model using the [MovieLens 100K dataset](https://grouplens.org/datasets/movielens/100k/) with TensorFlow Recommender System (TFRS) using Amazon SageMaker. \n",
    "\n",
    "We will use this model to recommend movies for a given user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q sagemaker==2.9.2\n",
    "!pip install -q sagemaker-experiments==0.1.24\n",
    "!pip install -q tensorflow==2.3.0\n",
    "!pip install -q tensorflow-recommenders==0.2.0\n",
    "!pip install -q tensorflow-datasets==4.0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify Input Data S3 URI and `Distribution Strategy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-1-835319576252/tensorflow_datasets/train/', 'S3DataDistributionType': 'ShardedByS3Key'}}}\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "input_train_data_s3_uri ='s3://{}/tensorflow_datasets/train/'.format(bucket)\n",
    "\n",
    "s3_input_train_data = TrainingInput(s3_data=input_train_data_s3_uri,\n",
    "                                    distribution='ShardedByS3Key')\n",
    "print(s3_input_train_data.config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Metrics To Track Model Performance\n",
    "\n",
    "These sample log lines...\n",
    "```\n",
    "499/500 [=====>..] - ETA: 3s - root_mean_squared_error: 1.194 - factorized_top_k/top_10_categorical_accuracy: 0.481 - factorized_top_k/top_50_categorical_accuracy: 0.607 - factorized_top_k/top_100_categorical_accuracy: 0.885\n",
    "```\n",
    "...will produce the following metrics in CloudWatch:\n",
    "\n",
    "`root_mean_squared_error` = 1.194\n",
    "\n",
    "`factorized_top_k/top_10_categorical_accuracy` = 0.481\n",
    "\n",
    "`factorized_top_k/top_50_categorical_accuracy` = 0.607\n",
    "\n",
    "`factorized_top_k/top_100_categorical_accuracy` = 0.885"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_definitions = [\n",
    "     {'Name': 'root_mean_squared_error', 'Regex': 'root_mean_squared_error: ([0-9\\\\.]+)'},    \n",
    "     {'Name': 'top_10_categorical_accuracy', 'Regex': 'factorized_top_k/top_10_categorical_accuracy: ([0-9\\\\.]+)'},\n",
    "     {'Name': 'top_50_categorical_accuracy', 'Regex': 'factorized_top_k/top_50_categorical_accuracy: ([0-9\\\\.]+)'},\n",
    "     {'Name': 'top_100_categorical_accuracy', 'Regex': 'factorized_top_k/top_100_categorical_accuracy: ([0-9\\\\.]+)'}\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Our TensorFlow Script to Run on SageMaker\n",
    "Prepare our TensorFlow model to run on the managed SageMaker service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Manual Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_variant='100k' # movielens 100k, 1m, 20m, 25m, etc\n",
    "enable_tensorboard=True\n",
    "train_instance_count=1\n",
    "train_instance_type='ml.p3.2xlarge'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "estimator = TensorFlow(entry_point='train.py',\n",
    "                       source_dir='src',\n",
    "                       role=role,\n",
    "                       instance_count=train_instance_count,\n",
    "                       instance_type=train_instance_type,\n",
    "                       py_version='py37',\n",
    "                       framework_version='2.3.0',\n",
    "                       hyperparameters={\n",
    "                           'dataset_variant': dataset_variant,\n",
    "                           'enable_tensorboard': enable_tensorboard\n",
    "                       },\n",
    "                       metric_definitions=metrics_definitions,\n",
    "                       debugger_hook_config=False\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Hyper-Parameter Ranges to Explore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimize the Model on SageMaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter\n",
    "from sagemaker.tuner import ContinuousParameter\n",
    "from sagemaker.tuner import CategoricalParameter\n",
    "from sagemaker.tuner import HyperparameterTuner\n",
    "                                                \n",
    "hyperparameter_ranges = {\n",
    "    'epochs': CategoricalParameter([1, 2]),\n",
    "    'learning_rate': ContinuousParameter(0.00001, 0.00005, scaling_type='Linear'),\n",
    "    'embedding_dimension': CategoricalParameter([32, 64]),    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = 'root_mean_squared_error'\n",
    "\n",
    "tuner = HyperparameterTuner(\n",
    "    estimator=estimator,\n",
    "    objective_type='Minimize',\n",
    "    objective_metric_name=objective_metric_name,\n",
    "    hyperparameter_ranges=hyperparameter_ranges,\n",
    "    metric_definitions=metrics_definitions,\n",
    "    max_jobs=6,\n",
    "    max_parallel_jobs=2,\n",
    "    strategy='Bayesian',\n",
    "    early_stopping_type='Off'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner.fit(inputs={\n",
    "            'train': s3_input_train_data\n",
    "          }, \n",
    "          include_cls_metadata=False,\n",
    "          wait=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'CreationTime': datetime.datetime(2020, 11, 29, 23, 2, 5, 672000, tzinfo=tzlocal()),\n",
      " 'HyperParameterTuningJobArn': 'arn:aws:sagemaker:us-east-1:835319576252:hyper-parameter-tuning-job/tensorflow-training-201129-2302',\n",
      " 'HyperParameterTuningJobConfig': {'HyperParameterTuningJobObjective': {'MetricName': 'root_mean_squared_error',\n",
      "                                                                        'Type': 'Minimize'},\n",
      "                                   'ParameterRanges': {'CategoricalParameterRanges': [{'Name': 'epochs',\n",
      "                                                                                       'Values': ['\"1\"',\n",
      "                                                                                                  '\"2\"']},\n",
      "                                                                                      {'Name': 'embedding_dimension',\n",
      "                                                                                       'Values': ['\"32\"',\n",
      "                                                                                                  '\"64\"']}],\n",
      "                                                       'ContinuousParameterRanges': [{'MaxValue': '5e-05',\n",
      "                                                                                      'MinValue': '1e-05',\n",
      "                                                                                      'Name': 'learning_rate',\n",
      "                                                                                      'ScalingType': 'Linear'}],\n",
      "                                                       'IntegerParameterRanges': []},\n",
      "                                   'ResourceLimits': {'MaxNumberOfTrainingJobs': 6,\n",
      "                                                      'MaxParallelTrainingJobs': 2},\n",
      "                                   'Strategy': 'Bayesian',\n",
      "                                   'TrainingJobEarlyStoppingType': 'Off'},\n",
      " 'HyperParameterTuningJobName': 'tensorflow-training-201129-2302',\n",
      " 'HyperParameterTuningJobStatus': 'InProgress',\n",
      " 'LastModifiedTime': datetime.datetime(2020, 11, 29, 23, 2, 5, 672000, tzinfo=tzlocal()),\n",
      " 'ObjectiveStatusCounters': {'Failed': 0, 'Pending': 0, 'Succeeded': 0},\n",
      " 'ResponseMetadata': {'HTTPHeaders': {'content-length': '3105',\n",
      "                                      'content-type': 'application/x-amz-json-1.1',\n",
      "                                      'date': 'Sun, 29 Nov 2020 23:02:05 GMT',\n",
      "                                      'x-amzn-requestid': 'e62fc70f-4f2b-4366-bb0d-6e3e7a5bc123'},\n",
      "                      'HTTPStatusCode': 200,\n",
      "                      'RequestId': 'e62fc70f-4f2b-4366-bb0d-6e3e7a5bc123',\n",
      "                      'RetryAttempts': 0},\n",
      " 'TrainingJobDefinition': {'AlgorithmSpecification': {'MetricDefinitions': [{'Name': 'root_mean_squared_error',\n",
      "                                                                             'Regex': 'root_mean_squared_error: '\n",
      "                                                                                      '([0-9\\\\.]+)'},\n",
      "                                                                            {'Name': 'top_10_categorical_accuracy',\n",
      "                                                                             'Regex': 'factorized_top_k/top_10_categorical_accuracy: '\n",
      "                                                                                      '([0-9\\\\.]+)'},\n",
      "                                                                            {'Name': 'top_50_categorical_accuracy',\n",
      "                                                                             'Regex': 'factorized_top_k/top_50_categorical_accuracy: '\n",
      "                                                                                      '([0-9\\\\.]+)'},\n",
      "                                                                            {'Name': 'top_100_categorical_accuracy',\n",
      "                                                                             'Regex': 'factorized_top_k/top_100_categorical_accuracy: '\n",
      "                                                                                      '([0-9\\\\.]+)'},\n",
      "                                                                            {'Name': 'ObjectiveMetric',\n",
      "                                                                             'Regex': 'root_mean_squared_error: '\n",
      "                                                                                      '([0-9\\\\.]+)'}],\n",
      "                                                      'TrainingImage': '763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.3.0-gpu-py37',\n",
      "                                                      'TrainingInputMode': 'File'},\n",
      "                           'EnableInterContainerTrafficEncryption': False,\n",
      "                           'EnableManagedSpotTraining': False,\n",
      "                           'EnableNetworkIsolation': False,\n",
      "                           'InputDataConfig': [{'ChannelName': 'train',\n",
      "                                                'DataSource': {'S3DataSource': {'S3DataDistributionType': 'ShardedByS3Key',\n",
      "                                                                                'S3DataType': 'S3Prefix',\n",
      "                                                                                'S3Uri': 's3://sagemaker-us-east-1-835319576252/tensorflow_datasets/train/'}},\n",
      "                                                'EnableFFM': False}],\n",
      "                           'OutputDataConfig': {'RemoveJobNameFromS3OutputPath': False,\n",
      "                                                'S3OutputPath': 's3://sagemaker-us-east-1-835319576252/'},\n",
      "                           'ResourceConfig': {'InstanceCount': 1,\n",
      "                                              'InstanceType': 'ml.p3.2xlarge',\n",
      "                                              'VolumeSizeInGB': 30},\n",
      "                           'RoleArn': 'arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881',\n",
      "                           'StaticHyperParameters': {'_tuning_objective_metric': 'root_mean_squared_error',\n",
      "                                                     'dataset_variant': '\"100k\"',\n",
      "                                                     'enable_tensorboard': 'true',\n",
      "                                                     'model_dir': '\"s3://sagemaker-us-east-1-835319576252/tensorflow-training-2020-11-29-23-02-03-891/model\"',\n",
      "                                                     'sagemaker_container_log_level': '20',\n",
      "                                                     'sagemaker_estimator_class_name': '\"TensorFlow\"',\n",
      "                                                     'sagemaker_estimator_module': '\"sagemaker.tensorflow.estimator\"',\n",
      "                                                     'sagemaker_job_name': '\"tensorflow-training-2020-11-29-23-02-03-891\"',\n",
      "                                                     'sagemaker_program': '\"train.py\"',\n",
      "                                                     'sagemaker_region': '\"us-east-1\"',\n",
      "                                                     'sagemaker_submit_directory': '\"s3://sagemaker-us-east-1-835319576252/tensorflow-training-2020-11-29-23-02-03-891/source/sourcedir.tar.gz\"'},\n",
      "                           'StoppingCondition': {'MaxRuntimeInSeconds': 86400}},\n",
      " 'TrainingJobStatusCounters': {'Completed': 0,\n",
      "                               'InProgress': 0,\n",
      "                               'NonRetryableError': 0,\n",
      "                               'RetryableError': 0,\n",
      "                               'Stopped': 0}}\n",
      "Not yet complete, but 0 jobs have completed.\n",
      "No training jobs have reported results yet.\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "tuning_job_name = tuner.latest_tuning_job.job_name\n",
    "\n",
    "job_description = sm.describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuning_job_name\n",
    ")\n",
    "\n",
    "status = job_description['HyperParameterTuningJobStatus']\n",
    "\n",
    "pprint(job_description)\n",
    "\n",
    "if status != 'Completed':\n",
    "    job_count = job_description['TrainingJobStatusCounters']['Completed']\n",
    "    print('Not yet complete, but {} jobs have completed.'.format(job_count))\n",
    "    \n",
    "    if job_description.get('BestTrainingJob', None):\n",
    "        print(\"Best candidate:\")\n",
    "        pprint(job_description['BestTrainingJob']['TrainingJobName'])\n",
    "        pprint(job_description['BestTrainingJob']['FinalHyperParameterTuningJobObjectiveMetric'])\n",
    "    else:\n",
    "        print(\"No training jobs have reported results yet.\")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/hyper-tuning-jobs/tensorflow-training-201129-2302\">Hyper-Parameter Tuning Job</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "    \n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/hyper-tuning-jobs/{}\">Hyper-Parameter Tuning Job</a></b>'.format(region, tuning_job_name)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".............................................................................................................................................................................................................................!\n",
      "CPU times: user 1.02 s, sys: 122 ms, total: 1.14 s\n",
      "Wall time: 18min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "tuner.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 9)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sagemaker.analytics import HyperparameterTuningJobAnalytics\n",
    "\n",
    "hp_results = HyperparameterTuningJobAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    hyperparameter_tuning_job_name=tuning_job_name\n",
    ")\n",
    "\n",
    "df_results = hp_results.dataframe()\n",
    "df_results.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# All Candidates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>embedding_dimension</th>\n",
       "      <th>epochs</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"64\"</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>0.000050</td>\n",
       "      <td>tensorflow-training-201129-2302-005-aec2a92a</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6797</td>\n",
       "      <td>2020-11-29 23:15:59+00:00</td>\n",
       "      <td>2020-11-29 23:19:43+00:00</td>\n",
       "      <td>224.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"64\"</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>tensorflow-training-201129-2302-006-e3e871b7</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6804</td>\n",
       "      <td>2020-11-29 23:16:01+00:00</td>\n",
       "      <td>2020-11-29 23:18:41+00:00</td>\n",
       "      <td>160.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"64\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>tensorflow-training-201129-2302-003-4457b41d</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6813</td>\n",
       "      <td>2020-11-29 23:10:50+00:00</td>\n",
       "      <td>2020-11-29 23:13:25+00:00</td>\n",
       "      <td>155.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"64\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>0.000043</td>\n",
       "      <td>tensorflow-training-201129-2302-004-89000343</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6815</td>\n",
       "      <td>2020-11-29 23:10:52+00:00</td>\n",
       "      <td>2020-11-29 23:13:23+00:00</td>\n",
       "      <td>151.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"64\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>tensorflow-training-201129-2302-002-621bcafa</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6826</td>\n",
       "      <td>2020-11-29 23:05:04+00:00</td>\n",
       "      <td>2020-11-29 23:07:44+00:00</td>\n",
       "      <td>160.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\"32\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>tensorflow-training-201129-2302-001-951e5119</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6887</td>\n",
       "      <td>2020-11-29 23:04:58+00:00</td>\n",
       "      <td>2020-11-29 23:07:39+00:00</td>\n",
       "      <td>161.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  embedding_dimension epochs  learning_rate  \\\n",
       "1                \"64\"    \"2\"       0.000050   \n",
       "0                \"64\"    \"2\"       0.000042   \n",
       "3                \"64\"    \"1\"       0.000048   \n",
       "2                \"64\"    \"1\"       0.000043   \n",
       "4                \"64\"    \"1\"       0.000022   \n",
       "5                \"32\"    \"1\"       0.000014   \n",
       "\n",
       "                                TrainingJobName TrainingJobStatus  \\\n",
       "1  tensorflow-training-201129-2302-005-aec2a92a         Completed   \n",
       "0  tensorflow-training-201129-2302-006-e3e871b7         Completed   \n",
       "3  tensorflow-training-201129-2302-003-4457b41d         Completed   \n",
       "2  tensorflow-training-201129-2302-004-89000343         Completed   \n",
       "4  tensorflow-training-201129-2302-002-621bcafa         Completed   \n",
       "5  tensorflow-training-201129-2302-001-951e5119         Completed   \n",
       "\n",
       "   FinalObjectiveValue         TrainingStartTime           TrainingEndTime  \\\n",
       "1               3.6797 2020-11-29 23:15:59+00:00 2020-11-29 23:19:43+00:00   \n",
       "0               3.6804 2020-11-29 23:16:01+00:00 2020-11-29 23:18:41+00:00   \n",
       "3               3.6813 2020-11-29 23:10:50+00:00 2020-11-29 23:13:25+00:00   \n",
       "2               3.6815 2020-11-29 23:10:52+00:00 2020-11-29 23:13:23+00:00   \n",
       "4               3.6826 2020-11-29 23:05:04+00:00 2020-11-29 23:07:44+00:00   \n",
       "5               3.6887 2020-11-29 23:04:58+00:00 2020-11-29 23:07:39+00:00   \n",
       "\n",
       "   TrainingElapsedTimeSeconds  \n",
       "1                       224.0  \n",
       "0                       160.0  \n",
       "3                       155.0  \n",
       "2                       151.0  \n",
       "4                       160.0  \n",
       "5                       161.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results.sort_values('FinalObjectiveValue', ascending=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Best Candidate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>embedding_dimension</th>\n",
       "      <th>epochs</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"64\"</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>tensorflow-training-201129-2302-005-aec2a92a</td>\n",
       "      <td>Completed</td>\n",
       "      <td>3.6797</td>\n",
       "      <td>2020-11-29 23:15:59+00:00</td>\n",
       "      <td>2020-11-29 23:19:43+00:00</td>\n",
       "      <td>224.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  embedding_dimension epochs  learning_rate  \\\n",
       "1                \"64\"    \"2\"        0.00005   \n",
       "\n",
       "                                TrainingJobName TrainingJobStatus  \\\n",
       "1  tensorflow-training-201129-2302-005-aec2a92a         Completed   \n",
       "\n",
       "   FinalObjectiveValue         TrainingStartTime           TrainingEndTime  \\\n",
       "1               3.6797 2020-11-29 23:15:59+00:00 2020-11-29 23:19:43+00:00   \n",
       "\n",
       "   TrainingElapsedTimeSeconds  \n",
       "1                       224.0  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results.sort_values('FinalObjectiveValue', ascending=1).head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow-training-201129-2302-005-aec2a92a\n"
     ]
    }
   ],
   "source": [
    "recommender_training_job_name = df_results.sort_values('FinalObjectiveValue', ascending=1).head(1)['TrainingJobName'][1]\n",
    "\n",
    "print(recommender_training_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pass to Next Notebook(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'recommender_training_job_name' (str)\n"
     ]
    }
   ],
   "source": [
    "%store recommender_training_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
